DOCUMENT RESUME 



ED 449 990 



SE 064 558 



AUTHOR 

TITLE 

PUB DATE 
NOTE 



PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



Costner, Kelly M. ; Wagner, Sigrid; Owens, Douglas T. 
Content-Responsive Program Evaluation: An Integrated 
Preservice Example . 

2000 - 02-00 

14p.; Paper presented at the Annual Meeting of the 
Association for Mathematics Teacher Education (Charlotte, 
NC, February 10-12, 2000). 

Information Analyses (070) -- Reports - Research (143) -- 

Speeches/Meeting Papers (150) 

MFOl/PCOl Plus Postage. 

Elementary Secondary Education; Evaluation Criteria; 
♦Evaluation Methods; Higher Education; *Mathematics 
Education; *Program Evaluation 



ABSTRACT 



This paper discusses the lack of and the need for program 
evaluation models for mathematics education and points out generic approaches 
to program evaluation. This project attempts to develop a mathematical 
approach to program evaluation that can be applied to an integrated 
preservice program for mathematics, science, and technology education 
teachers. Guiding questions for the overall project include: (1) How can 

program evaluators approach their task in a way that is valid for mathematics 
specialists; specifically, how can program evaluation be approached in a way 
that reflects the values of the mathematics education community? and (2) What 
is the role of the evaluator in content- responsive program evaluation? 
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In an age of accountability it becomes increasingly important to document what 
we do and how well we do it. Program evaluation provides a comprehensive view of 
the effectiveness of a newly implemented (or well-established) curriculum or other 
program, and is a means of providing the accountability that is often expected by 
administrators and the public. Program evaluation can also provide data useful in the 
successful propagation of reform efforts, whether they are at the school, higher 
education, or pre- and inservice teacher education levels. 

The Problem: A Lack of Program Evaluation Models 
Specific to Mathematics Education 

As mathematics educators — content specialists dedicated to the subject we 
teach — we would naturally look for models of program evaluation within our field. 
Alas, there are few. Indeed, a search of related literature revealed only three "how-to" 
guides on program evaluation in mathematics. 
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Mathematics Program Evaluation 

The first such guide is a publication of the National Council of Teachers of 
Mathematics (NCTM) titled How to Evaluate Your Mathematics Program (1981). This 19- 
page booklet consists of a set of 21 standards for school mathematics programs 
established by a collaborative in Maryland, and grouped into the following categories: 
instruction, curriculum and instructional materials, the teacher, and physical facilities 
and equipment. Since this document was published several years earlier than NCTM's 
Curriculum and Evaluation Standards (1989) and the ensuing reform efforts, a current 
program would have somewhat different goals and objectives from those indicated in 
these standards. 

The second mathematics-specific guide is a joint project of NCTM and the 
Association for Supervision and Curriculum Development, titled A Guide for Reviewing 
School Mathematics Programs (1991). This document is more extensive in scope, 
including curricular goals by grade levels and aspects of school administration, but 
remains little more than a survey-type instrument. In general this 65-page document is 
of limited use in investigating the interrelatedness of factors that results in the success 
(or failure) of a program. 

The final mathematics-specific guide is also from the Association for Supervision 
and Curriculum Development, the Mathematics Assessment Process: A Curriculum 
Alignment Strategy (1992). The result of an extensive project with national consultants, 
this two-volume publication takes the idea of program evaluation much further. This 
assessment process differs from the other two in that its stated purpose is to assist in 
making changes in a mathematics program to help the program become more aligned 
with the NCTM Standards. Thus, this mathematics program evaluation is intended to 
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do more than assess the current status of a program, and is in fact designed to help 
guide a program toward very specific goals. 

In summary, none of these examples provides a comprehensive guide for overall 
evaluation of mathematics programs. They are either limited in scope, in evaluation 
activities, or in grade level applicability, and essentially provide instruments rather than 
an approach or model. Lacking a specific model for mathematics program evaluation, 
we turn naturally to the more general program evaluation literature. 

Generic Approaches to Program Evaluation 

Approaches to program evaluation abound, and are categorized differently by 
various researchers. One of the most commonly known distinctions is that of 
summative and formative evaluation as first delineated by Scriven (1967). Summative 
evaluation occurs after a program is completed and is used to determine the program's 
effectiveness, while formative evaluation occurs while a program is in progress and is 
used to change or further develop the program in some way. 

Fitz-Gibbon and Morris (1987) present a very positivistic approach to program 
evaluation. They offer six ways to design an evaluation: (1) true control group, pretest- 
posttest; (2) true control group, posttest only; (3) nonequivalent control group pretest- 
posttest; (4) single group time series; (5) time series with nonequivalent control group; 
and (6) before and after. These designs assume an experimental orientation to program 
evaluation, and can only be effectively implemented in nearly ideal conditions. 

Stecher and Davis (1987) as well as Worthen, Sanders, and Fitzpatrick (1997) go 
beyond these approaches to present a continuum from more positivistic to more 
naturalistic orientations. The objective-oriented approach emphasizes the goals and 
objectives of the program and attempts to measure the success of the program in terms 
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of its goals. In this type of evaluation, the evaluator takes on the role of measurement 
specialist. 

The management-oriented approach is guided by the decision needs of the 
program — data are gathered in relation to issues on which changes are already being 
contemplated or at crucial steps in implementation or progress. Effective program 
management is a goal, and the evaluator must work backwards from clear decision 
points to determine what information is needed to inform the decision-making process. 

The consumer-oriented approach concentrates on providing information that will 
be useful to those who are in a position to take action based on the data. The emphasis 
is on people, and the evaluator involves user groups throughout the evaluation. 

The participant-oriented approach puts the evaluator in the position of seeking 
understanding of all evaluation questions from the point of view of all participants, 
program staff, administrators, and others. This approach uses more qualitative data 
collection methods, and therefore is functional in less focused evaluations. 

The expertise-oriented approach is the term Worthen et al. (1997) use to denote the 
oldest and perhaps best known form of evaluation, in which a professional in the 
particular field is called upon to evaluate the specific program. In this approach, an 
expert or a team of experts in the relevant field is called upon to provide professional, 
subjective judgements about the program being evaluated. A review for accreditation is 
one example that would fall under this category of evaluation. Adversary-oriented 
approaches make up the other category offered by Worthen et al. (1997). This approach 
borrows ideas from the field of law, in that the evaluation plan attempts not only to 
generate but also balance opposing viewpoints on a program. This process often 
involves conducting open "hearings" in which all viewpoints are freely expressed. 
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Worthen et al. also provide a chapter in which they evaluate the contributions of 
the six categories of approaches presented. They point out that program evaluation is a 
relatively new field, and that it is not considered yet to be a "full-grown discipline" (p. 
172). They continue by stating that the approaches presented do not meet the criteria to 
be considered scientific models. Further, these categories of approaches are not 
theories, since they are not nor can they be written in axiomatic style, and they do not 
allow us to make predictions or conjectures of any sort. They conclude that these 
approaches are simply useful information. Indeed, they point out that the field 
continues to expand, and that it will be much longer before we can attempt to put 
together all we know about program evaluation into a model. 

The Accompanying Paradigm Shift 

The varying approaches to evaluation in fact reflect a philosophical shift that has 
been underway for some time in the field of program evaluation. Greene (1994) 
describes the evolution of program evaluation in terms of an underlying move through 
several philosophical paradigms. Greene essentially explains that the goal-oriented 
approach reflects a postpositivist philosophical framework; the decision-oriented 
approach shows a pragmatic viewpoint; and the user-oriented and responsive 
approaches demonstrate an interpretivist framework. 

Greene (1994) mentions several program evaluation researchers who have made 
significant contributions toward the paradigmatic shift. Stake's responsive evaluation 
(1975, as cited in Greene, 1994) was one model that helped move program evaluation 
toward the interpretivist paradigm. The responsive approach is focused on identifying 
and addressing the concerns of all the stakeholders in a program, and examines those 
concerns in terms of program improvement. 
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Another milestone in the evolution of program evaluation was, according to 
Greene (1994), Eisner's acknowledgement of the evaluator in the evaluation (1991, as 
cited in Greene, 1994). Eisner's approach focuses on interpretation and judgement by a 
connoisseur — an expert in the field of the program being evaluated. 

Greene further explains that we have moved beyond interpretivism to an era 
when critical theory is being applied to program evaluation, specifically through the 
work of Guba and Lincoln (1989, as cited in Greene, 1994) in what they call fourth- 
generation evaluation. Greene explains that this approach uses a constructivist 
framework, which "requires that evaluation catalyze social action" (p. 540). If indeed 
the content-responsive model is to focus on some aspect of mathematics program 
improvement,. it is to do so with the expectation that those engaged in the program will 
in fact improve it — which we believe may be the type of social action intended by Guba 
and Lincoln. Indeed, Greene describes the role of the evaluator in fourth-generation 
evaluation as one who facilitates social change, a role similar to that we envision for the 
evaluator in content-responsive evaluation. 

These three approaches — Stake's responsive approach, Eisner's connoisseur- 
evaluator approach, and Guba and Lincoln's fourth-generation approach — will be the 
focus of our endeavor to create the content-responsive approach to program evaluation. 
Need for a Mathematics-Specific Approach: What Hasn't Worked for Mathematics 

The need for a mathematics-specific approach arises from concerns that are often 
expressed by mathematics specialists when program evaluations are undertaken. One 
problem is that the program evaluator is often an outsider — not a mathematician. 
Lacking knowledge of mathematics itself and not sharing the values and culture of 
mathematics education, the evaluator tends to look at more general questions, 
concentrating on the success or failure of documented forms of goals and focusing 




7 



7 



perhaps on resources (supply and demand issues). Another problem with 
nonmathematical program evaluation is that too great an emphasis is put on what we 
might call "appeal" questions. That is, there may be a focus on issues of satisfaction, 
which might translate, for example, to an analysis of levels of mathematics anxiety 
among students. Similarly, there can be a focus on reform issues only (rather than an 
overall view of the program), especially at a time when everyone is clamoring to prove 
that programs are in fact reform oriented. This fact calls for an approach to evaluation 
that we as mathematics specialists can undertake ourselves, and one in which we can 
have faith as mathematical thinkers. Furthermore, a mathematics specialist will know 
how to make specific recommendations for program improvement, thereby increasing 
the likelihood that an evaluation report will be used. 



Research Questions 

This project attempts to develop a mathematical approach to program 
evaluation, which will then be applied to an integrated preservice program for 
mathematics, science, and technology education teachers. Guiding questions for the 
overall project are as follows: 

1. How can program evaluators approach their task in a way that is valid for 
mathematics specialists? Specifically, how can program evaluation be approached 
in a way that reflects the values 'of the mathematics education community? 

2. What is the role of the evaluator in content-responsive program evaluation? 

Developing the Content-Responsive Approach 
In this section we will explore the way that we propose to apply mathematics to 
the field of evaluation, thus creating a mathematically sound model for evaluation of 
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mathematics and/or mathematics education programs. Also, we will discuss several 
specific models from the generic program evaluation literature that will require further 
examination in the development of the content-responsive model. 

The Metaphor in Mathematics 

Both Lynch (1996) and Worthen et al. (1997) mention the contributions of Smith 
(1981), who writes of various metaphors that have or can be applied to the field of 
evaluation. For instance, the adversary-oriented approach mentioned earlier was 
developed using the metaphor of a trial or hearing, with advocates for multiple 
perspectives presenting their cases to a jury of sorts. Other metaphors abound, 
including ideas from fields such, as investigative reporting, architecture, geography, 
philosophy, literary and film criticism, and watercolor painting (Smith, 1981). - 

Problem Solving. Given that we are searching for an approach suitable to the 
field of mathematics, and one that would be viewed as credible by content specialists in 
mathematics, it would seem appropriate to identify a mathematical metaphor for 
evaluation. The metaphor we propose is that of problem solving — a nonalgorithmic 
approach to mathematical tasks, and one that seems to have relevance to evaluation 
functions. Certainly, this metaphor is accessible and credible to mathematicians. 

Problem-solving literature in mathematics education generally points to one 
common conception of problem solving: the procedure offered by Polya (1957) in the 
oft-cited volume How to Solve It. Specifically, Polya identifies four stages of problem 
solving: (a) understanding the problem, (b) devising a plan; (c) carrying out the plan, 
and (d) looking back to review the procedure and outcome (pp. xvi-xvii). These stages 
have direct application to the program evaluation process, and indeed, imply that that 
the evaluation of a program is in fact a problem to be solved. 




9 



9 



Understanding the problem. First, we must understand the problem of 
program evaluation. The tradition of program evaluation is that the evaluation report 
gives a pronouncement as to whether a program is successful or not, and makes some 
recommendations for improvement based on that fact. The focus of the evaluation 
process, however, is on determining how well the program is achieving its goals — the 
recommendations come as a result of that assessment. Indeed, it seems that the 
problem being addressed by typical program evaluations can be phrased as a question: 
"Is this program good?" We propose that the question be changed, that we instead 
begin to focus on a question that presents a real problem: "How can this program be 
improved?" This question gets more directly at the point of program evaluation in the 
first place, but it goes beyond just that by streamlining the evaluation process. That is, 
rather than first determining the worth of a program and then developing 
recommendations for improvement, the application of the problem solving process to 
this new question puts the evaluator and participants in a new position, with a different 
stance. Indeed, a focus on improvement automatically shifts the evaluator and 
participants into a problem-solving mode. Having developed this new understanding 
of the problem of program evaluation, we must look at the next of Polya's stages and 
devise a plan for solving the problem of program improvement. 

Devising a plan. Many mathematicians and educators have developed problem- 
solving strategies that lead the problem solver through Polya's stages. For example, 
Posamentier and Krulik (1998) present the following heuristics for devising a plan: 
working backwards, finding a pattern, adopting a different point of view, solving a 
simpler (analogous) problem, considering extreme cases, intelligent guessing and 
testing, organizing data, and using logical reasoning. Such strategies are frequently 
mentioned in the mathematics education reform documents (including the NCTM 
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Standards [1989]) that are currently calling for a problem-solving emphasis in the 
teaching and learning of mathematics. The problem-solving vocabulary is a part of the 
mathematics educator's everyday language; problem-solving skills are what we address 
in our classrooms every day, as well. In terms of devising a plan, then, it seems logical 
to apply these very strategies (with which we are so familiar) to the problem of program 
improvement. 

Some problem-solving strategies such as organizing data and working 
backwards are fairly obvious aspects of any evaluation. However, the direct 
application and open acknowledgement of these processes would contribute to a more 
active view of program evaluation, and would greatly streamline the evaluation 
process. 

Carrying out the plan. Another key aspect of the application of problem solving 
to contribute to a more active view of the evaluation process is the fact that problem 
solving includes the act of solving a problem — that is, a plan is designed and it is carried 
out. In the content-responsive model proposed here, the carrying out of the plan means 
utilizing the heuristics in ways that will reveal useful information about the program 
and lead to ideas for program improvement. It is at this point that the role of the 
evaluator becomes key. Should the use of the heuristics — that is, the application of the 
working backwards procedure, or the adoption of a different point of view — be entirely 
up to the evaluator, entirely up to the program administrators, or some combination of 
these two? On the one hand, if the evaluator alone takes on this part of the process, the 
evaluation would end up looking more like one using the expertise-oriented approach. 
On the other hand, if left solely to the program administrators, the evaluation would 
lose the valued insight of the mathematics specialist in the evaluator role — a point that 
has been argued throughout this paper. 
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Looking back to review procedure and outcome. This stage of Polya's problem 
solving process involves evaluation of the outcome and of the process used to reach that 
outcome. As applied to program evaluation, this will involve looking to see if the use 
of problem-solving heuristics produced actual program improvements. That is, the 
evaluator and program administrators will review the plan that was designed to 
improve the program, asking questions about its efficiency and productivity, and will 
examine the changes implemented to determine if improvements did in fact result. 

Overall, the metaphor of problem solving should contribute significantly to the 
field of mathematics education and that of program evaluation itself. Specifically, if 
program evaluators work with program administrators and other stakeholders to solve 
the problem of program improvement in the context of the program at hand, then the 
evaluation process will come to be viewed as a more positive and productive 
experience — a mathematically sound venture that effects productive change. 
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